A Corpus based Probabilistic Grammar with Only Two Non terminals

The availability of large syntactically bracketed corpora such as the Penn Tree Bank a ords us the opportunity to automatically build or train broad coverage grammars and in particular to train probabilistic grammars A number of recent parsing experiments have also indicated that grammars whose production probabilities are dependent on the context can be more e ective than context free grammars in selecting a correct parse To make maximal use of context we have automatically constructed from the Penn Tree Bank version a grammar in which the symbols S and NP are the only real non terminals and the other non terminals or grammatical nodes are in e ect embedded into the right hand sides of the S and NP rules For example one of the rules extracted from the tree bank would be S NP VBX JJ CC VBX NP where NP is a non terminal and the other symbols are terminals part of speech tags of the Tree Bank The most common structure in the Tree Bank associated with this expansion is S NP VP VP VBX ADJ JJ CC VP VBX NP So if our parser uses rule in parsing a sentence it will generate structure for the corresponding part of the sentence Using of the Penn Tree Bank for training we extracted distinct rules for S and for NP We also built a smaller version of the grammar based on higher frequency patterns for use as a back up when the larger grammar is unable to produce a parse due to memory limitation We applied this parser to Wall Street Journal sentences separate from the training set and with no limit on sentence length Of the parsed sentences the percentage of no crossing sentences is and Parseval recall and precision are and